Skip to content

SQL Subquery Decorrelation - #38929

Closed
damccorm wants to merge 3 commits into
apache:masterfrom
damccorm:feature/sql-decorrelation
Closed

SQL Subquery Decorrelation#38929
damccorm wants to merge 3 commits into
apache:masterfrom
damccorm:feature/sql-decorrelation

Conversation

@damccorm

Copy link
Copy Markdown
Contributor

Adds a custom subquery decorrelation pass to Beam SQL to handle complex correlated subqueries.

The Problem

Calcite's default SqlToRelConverter decorrelates most queries, but some complex correlated shapes (like correlated EXISTS or IN subqueries inside PROJECT or JOIN conditions) survive as RexSubQuery or residual LogicalCorrelate nodes.

Beam's Volcano planner ruleset has no physical converter rules for a general LogicalCorrelate node. When these residues survive, the planner fails with a CannotPlanException.

The Fix

Implemented a pre-Volcano normalization pass in CalciteQueryPlanner.convertToBeamRel:

  1. normalizeForVolcano: This pass runs a short-lived HepPlanner with Calcite's *SUB_QUERY_TO_CORRELATE rules to turn any un-expanded RexSubQuery into a LogicalCorrelate.
  2. It then calls RelDecorrelator.decorrelateQuery to lower these correlates into standard relational shapes (Join, Aggregate, Project, Filter) which existing Beam rules already know how to handle.
  3. This pass is strictly gated on the tree actually referencing a correlation variable (using RelOptUtil.getVariablesUsed), making it a no-op for shapes like UNNEST (LogicalCorrelate(_, Uncollect)) which define but do not reference a correlation variable, preserving them for BeamUnnestRule.
  4. It runs before the Volcano planner and cost-based metadata provider swap, keeping it off the recursive cost path.

Added a detailed design document in decorrelation_design.md explaining the approach and the Volcano planner convergence safety.

@codecov

codecov Bot commented Jun 11, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 37.30159% with 79 lines in your changes missing coverage. Please review.
✅ Project coverage is 54.32%. Comparing base (d9117f4) to head (27b2d0b).
⚠️ Report is 55 commits behind head on master.

Files with missing lines Patch % Lines
...m/sdk/extensions/sql/impl/CalciteQueryPlanner.java 36.36% 64 Missing and 6 partials ⚠️
...pache/beam/sdk/extensions/sql/impl/BeamSqlEnv.java 30.00% 7 Missing ⚠️
...che/beam/sdk/extensions/sql/impl/QueryPlanner.java 0.00% 2 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #38929      +/-   ##
============================================
- Coverage     57.77%   54.32%   -3.46%     
+ Complexity    12969     1707   -11262     
============================================
  Files          2509     1062    -1447     
  Lines        260525   166535   -93990     
  Branches      10658     1252    -9406     
============================================
- Hits         150516    90462   -60054     
+ Misses       104318    73860   -30458     
+ Partials       5691     2213    -3478     
Flag Coverage Δ
java 67.05% <37.30%> (+2.87%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@damccorm damccorm closed this Jun 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant